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(57) Abstract: Methods and apparatus for increasing resolution in digital imaging systems reposition the fixed detector arrays (or 
the red, green, and blue images on the detector arrays) so that the resolution of the composite three color image is increased. For 
sample, the green CCD image is shifted horizontally and vordcally 1/3 of a pixel with respect to the red CCD image, and the 
blue CCD image is further shified horizontally and wrucafly 1/3 of a pixel with respect to the green CCD image. Image alignment 
may ae accomplished with some <*rnWrratSon 0 f CCD movement and/or image movement The color recovery process, also called 
reconstruction, may be aided by use of one of several interpolation methods. Color filter arrays may be used over the CCDs. 
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METHODS AMn APPARATUS FOR INCREA SING RESOU mON IN DIGITAL 
IMAGING SYSTE MS RY OFFSETTING PIXELS 

This Invention was made with Government support awarded by the 
national science foundation under Grant Number NSF M1P-9613754 and 
the Army Research Office under Grant Number DOD DAAG 55-97-1- 
0348. The Government has certain rights in this invention. 

BACKGROUND OF THE INVENTION 

FIELD OF THE INVENTION 

The present invention relates to methods and apparatus for increasing 
resolution in digital imaging systems by permanently offsetting pixels 
in overlapped red, green and blue images and reconstructing the full 
color image. 

DESCRIPTION OF THE PRIOR ART 

In digital cameras, there are two generally accepted methods for 
acquiring color images using a light detector array. The first method 
is to put a color filter array (CFA) over the detector array, such that 
certain pixels detect certain colors. For example, one third of the 
pixels would detect red light, one third of the pixels would detect green 
light, and one third of the pixels would detect blue light. The result of 
using such a color filter is that each pixel in the array only sees one 
color, so that to obtain a full color image, interpolation must be used 
to estimate the other two colors at each pixel. Figures 1A and 1B 
(prior art) show how this method is used. The image obtained by the 
detector array through the color filter array has an arrangement of 
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red pixels, blue pixels, and green pixels, for example in the Bayer CFA 
pattern, as shown in Figure 1A. The intensity of each pixel indicates 
how much of the selected color of light was detected. In Figure 1B all 
of the pixels have all three colors of light, because the green light is 
interpolated in some manner, for example between pixels G1, G4, G5, 
and G7 to determine the intensity of green light in the upper left hand 
pixel , and so on. Thus, each three color pixel is really accurate only 
with respect to one of the three colors. 

The second method is more expensive, and more accurate with respect 
to color. Three separate detector arrays are used, one for each color. 
Thus three images, one for each color, are formed. Then the three 
images are overlapped to give a composite full color image. The grey 
scale resolution is no better than the first method, but the accuracy 
of the colors is greater. Figure 2 (prior art) indicates how the three 
Images are overlapped to generate an accurate three color image. In 
addition, each array has a 100% fill factor (ratio of pixel size to pixel 
spacing), meaning there is little or no dead space between pixels. This 
reduces aliasing. Figure 3 (prior art) shows how this occurs. The main 
lobe of the image spectrum ourve is as narrow as possible (the first 
zero is at the sampling frequency) when the firl factor is 100%. 

Aliasing is an important factor in generating color images. Aliasing 
occurs when the lens presents more spatial detail to the detector 
array than it can record. The spatial detail that is left over appears as 
incorrect, less detailed information. That is, the image has errors 
caused by the detailed information that is masquerading, or aliasing, as 
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less detailed information. 

It is known in the art that the resolution of a detector array can be 
increased by making multiple exposures of the image and moving the 
image with respect to the detector array between exposures. For 
example, if the detector array is shifted by 1/2 pixel, the image 
resolution recorded by the combination of the two images is doubled in 
the direction of the shift. This gives erroneous images if the object 
moves between exposures. For example, a black on white edge would 
appear as a colored edge. 

A need remains in the art for methods and apparatus for increasing 
resolution and reducing aliasing in digital imaging systems by 
permanently offsetting pixels in overlapped red, green and blue images 
and properly reconstructing the full color image- 
Si 1MMARY OF THE INVENTION 

It is an object of the present invention to provide methods and 
apparatus for increasing resolution in digital imaging systems by 
permanently offsetting pixels In overlapped red, green and blue images 
and properly reconstructing the full color image. 

This is accomplished by repositioning the fixed detector arrays (or the 
red, green, and blue images on the detector arrays) so that the 
resolution of the composite three color image is increased. For 
.example, the green CCD image is shifted horizontally and vertically 1/3 
of a pixel with respect to the red CCD image, and the blue CCD image is 
further shifted horizontally and vertically 1/3 of a pixel with respect 
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to the green CCD image. The detectors themselves need not be 
moved, but instead the three images can be shifted by the appropriate 
subpixel amount. The color representation of the resulting image can 
be as good as the image acquired with the detector arrays aligned, and 
the resolution can be neariy tripled (the fill factor must be decreased 
to triple the resolution). 

In the general case, n arrays are used. Each array has pixels of height 
p and width a, and each one Is shifted by p/n vertically and/ or cy/n 
horizontally. 

Another embodiment uses three detector arrays, each with a color 
filter array on It, and displaces the images by subpixel amounts as 
described above. One advantage of this approach is that some color 
filter arrays allot proportionally more pixels to green, because it is 
more important to human viewers. An example of a color filter array 
rs the Bayer pattern color filter array, shown in Figure 1 (prior art). 

The color recovery process, also called reconstruction, is aided by use 
of one of several interpolation methods. The simplest method entails 
replicating the color value of each pixel nine times (in a 3 by 3 array) 
and using the raw red, green, and blue values at each effective pixel 
location. 

A second reconstruction method entails using simple linear 
interpolation of each color plane. A third reconstruction method 
interpolates the green color plane by means of linear interpolation, and 
then interpolates the red and blue color planes with help from the 
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knowledge of the green color plane. FEB 2 8 2007 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1A (prior art) shows an image obtained by a detector array 
through the color filter array. 

Figure 1B (prior art) shows the reconstructed three color image with 
one method for interpolating the pixels of the image of Figure 1A, 

Figure 2 (prior art) indicates how the three one color images are 
overlapped to reconstruct a three color image. 

Figure 3 (prior art) shows the image spatial frequency spectrum curve 
obtained using the method of Figure 2. 

Figure 4 is a diagram showing red, green, and blue CCD images with 
offset pixels according to the present invention. 

Figure 5 shows the image spectrum curve obtained using the offset 
pixel method of Figure 4. 

Figure 6A is a diagram showing three images obtained by detectors 
through Bayer color filter arrays with offset pixels. Figure 6B is a 
diagram showing a similar two detector embodiment. 

Figure 7 shows a three by three array of reconstituted pixels obtained 
by a first reconstruction method according to the present invention. 

. Figure 8A is a diagram showing a blue array with zero padding and 
Figure 8B shows a low pass filter for filtering this array illustrating a 
second reconstruction method according to the present invention. 
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Figure 9 Is a flow diagram illustrating a third reconstruction method 
according to the present invention. 

DETAILED DESCRIPTION Q ETHR PREFERRFD FMBQDfMENIS 

Figure 4 is a diagram showing red, green, and blue CCD images with 
offset pixels according to the present invention. These CCD images 
are shown as arrays having three by three pixels, for simplicity. In 
practical use, the CCD images would be on the order of 500 by 500 
pixels, or larger. 

in the example of Figure 4, red CCD image 402 is overlaid by blue CCD 
image 404, which in turn is overlaid by green CCD image 406. In one 
preferred embodiment, blue CCD image 404 is offset by 1/3 of a pixel, 
both vertically and horizontally, from red CCD image 402. Green CCD 
image 406 is further offset by 1/3 of a pixel both vertically and 
horizontally, in the same direction. This orientation results in an 
effective sample rate that is three times the normal sampling rate, 
vertically and horizontally, of the individual CCD arrays. The effective 
sample rate around the perimeter of the overlaid image is not as high, 
but in most imaging systems, there are surplus pixels around the 
perimeter of the array detector. 

Each sensor, or CCD, has its own inherent resolution cutoff, which is 
affected by both the pixel spacing and the pixel size. The digital signal 
processing combines the three images from the CCDs and forms the 
effective pixels, reconstructing the image, Figure 7 shows the 
simplest method of reconstruction in the embodiment of Figure 4. 

6 
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In the general case, n arrays could be used. Each array has pixels of 
height p and width o\ and each one is shifted by p/n vertically and/or 
cy/n horizontally. The detected image in the horizontal dimension, y h > 
can then be written as: 



y h - fe(x) *tect(7)] 3 combC-^-) (1) 



i-0 



Where g(x) is the intensity distribution at the detector, * indicates the 
convolution operator, 

i_ 4 s(x - k) 

Taking the Fourier transform of (1) the spectrum of the detected 
image is 

n-1 

3 

1»0 



y h =o 2 G(u)sinc(au)*comb(ou) 3 exp(-j2jtuj^) (2) 



The summation in (2) can be rewritten as, 



§xp(-jTO { n } ) 3 exp(jrai-^-(2i+l)),nevcn 



3 ap(-JM T ) = | (B . I)/a W 

uo 1 exp(-jrcu (n ~ 1)g ) 3 exp(j?tu-|-2i),nodd 

5 a l=-(n-l>/2 



Using the Euler relation the summations in (3) can be written as 
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cosines as: 

n/4 



fxp(-j7tu^^) 3 2cos(2™-2-(i+-^),neveii 

— 1 <s „ s n 5=10 (4) 

3 e xp(^i T )=g ' 



< 



3 2cos(2rcu-jp) 



Two cases of particular interest are when n-2 and n=3. Ignoring the 
phase term, which is not detected by a CCD, the image spectra of 
these two cases are: 

y h | o=a = o l G(u)sinc(<yu)*2oos(27ro7)comb(cru) (5a) 

y* " C J G(u)sinc(OTi)*(l +2cos(2im|-))coinb(<yii) (5b) 

Figure 5 shows the image spectrum curve obtained using the offset 
pixel method of Figure 4. 

Curve 506 is the closest replicated spectrum (caused by the sampling) 
that remains after sampling with a 1/2 pixel shift. For example, the 
blue image may be moved down 1/2 pixel and the green image may be 
moved to the side 1/2 pixel. 

Curve 508 is the closest replicated spectrum that remains after 
sampling with a 1/3 pixel shift, and shows that again there is no 
aliasing. In this case the blue image might be moved down 1/3 pixel and 
to the right 1/3 pixel, and the green image might be moved down 2/3 
pixel and to the right 2/3 pixel. Curve 510 is a sine function related 
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to the width of the pixel, and provides low pass filtering of the image. 
To increase the resolution in the case of a 1/3 pixel shift, the pixel 
size (and hence the fill factor) must be reduced. This broadens the 
sine function and thereby passes higher frequencies. . Any replicated 
spectrum that overlapped the image spectrum would result in aliasing 
of the image. Comparison with the image spectrum of Figure 3 (prior 
art) shows how the resolution has increased, and aliasing has been 
greatly reduced or eliminated. 

Figure 6A is a diagram showing three images obtained by detectors 
through Bayer color filter arrays with offset pixels, Bayer color filter 
arrays (CFAs) use twice as many green pixels as red or blue pixels, 
because green is the most important color to human viewers. The 
three images (for example) are overlaid in an offset manner as 
described above. There are a multitude of variations on this idea not 
only in reconstruction methods but also in the CCD CFAs used. These 
CFAs take on many different forms and are usually designed to 
facilitate reconstruction through knowledge of the human visual 
system. The idea of using shifted CCD arrays to increase resolution, 
antialiasing and color performance can be extended to use any 
combination of CFAs. Another such example might be the use of one 
green CCD and two Bayer CFAs. 

Figure 6B shows an alternative embodiment, wherein one detector 
array could be used to detect the green portion of images, and a 
second array with red and blue filters could be displaced 1/2 pixel both 
in the x and y directions. This doubles the resolution of the system, 

9 
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reduces aliasing, provides more green information for the color 
interpolation (or image reconstruction), and reduces the number of 
CCD arrays needed. 

Given the 1/3 pixel shifting described with regard to Figure 6A, a 
generalized pixel results in nine effective pixels, as shown in Figure 4. 
Figure 7 shows a three by three array of reconstituted effective pixels 
obtained by the simplest reconstruction method (for example section 
408 of Figure 4, coterminous with blue pixel 22). This method of 
reconstruction entails replicating the value of each color pixel nine 
times (in a three by three array), and applying the raw red, green, and 
blue values at each location. 

The present method does not attempt to increase the resolution of 
each channel, but rather that of the final three-color image. In order 
to prevent unwanted color effects in the final image, some form of 
reconstruction is desirable. Figures 8a and 8b illustrate linear 
interpolation of each channel, or color plane. 

Figure 8A is a diagram showing a blue array with zero padding. Each 
channel or color plane is increased to nine times its original size (e.g. A 
100 by 100 pixel array would be transformed into a 300 by 300 value 
array) by inserting eight zeroes around each pixel value. A simple low 
pass filter is then applied to each channel in order to replace the 
inserted zeroes with their estimated values as shown in Figure 8B. 

Figure SB shows a five by five low pass filter for filtering the array of 
Figure 8A (the filter can also be larger). The coefficients used in the 
filter are denoted u, w, x, y, z and can be optimized for best 

10 
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performance by those skilled in the art of filter design. The only 
restriction on the coefficients of the filter shown in Figure 8B are that 
x+z=1 and 2w+y+u-1 . This reconstruction method produces a better 
image than the method shown in Figure 7, but it ignores the fact that 
the human eye responds better to green light than to red or blue. 
Also, it blindly interpolates each channel separately. 

Figure 9 is a flow diagram illustrating a third reconstruction method, 
which interpolates the green channel first, and then uses the results 
to better interpolate the red and blue channels. This method avoids 
the blind interpolation of the previous method and also utilizes a major 
aspect of the human visual system. In the case of the blue channel, 
the above objective is achieved as illustrated in Figure 9. Again, eight 
zeros are placed around the known blue values to create a zero-padded 
veision of the blue channel as shown in Figure 8A. The zero-padded 
blue channel and the interpolated green channel are divided on a pixel 
by pixel basis. This yields a new channel that has ratios of blue and 
green at pixel locations where blue was known and zeros elsewhere. It 
is this new channel that is then lowpass filtered, recalling the same 
nuances described in the method above, to replace the zero values. 
Finally, a pixel by pixel multiplication with the resulting channel and the 
interpolated green channel produces the final interpolated blue channel. 
An identical process is used to interpolate the red channel. The three 
interpolated channels now compose a reconstructed image with 
increased resolution, antialiasing and color performance. 

Those skilled in the art will appreciate that many other configurations 

11 
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fall within the spirit and scope of the present invention. 
What fe claimed is: 



12 



PAGE 22/1 27 * RCVD AT 2/28/2007 4: 00:28 PM [Eastern Standard Time] * SVR:USPT0-EFXRF-1/1 9 * DNIS:2738300 * CSID:16508575487 * DURATION (mm-ss):33-00 



FE8. 28. 2007 1:02PM HP LEGAL IP 

WO 01/99431 



NO. 111 P. 23 

PCTAJS01/18954 



CLAIMS 

1 . Improved apparatus for increasing resolution in a digital imaging 
system of the type having at least two fixed detector arrays for 
imaging an object and reconstruction means for composing a 
composite image from the images captured by the detector arrays, 
wherein the improvement comprises: 

each detector array is located such that its captured image is shifted 
by a subpixel amount with respect to the images captured by the other 
detector arrays; and 

the reconstruction means provides color for each portion of the 
shifted images. 

2. The apparatus of claim 1, having two fixed detector arrays, one- 
configured for capturing a red and blue image, and the other for 
capturing a green image, wherein the images captured by the detector 
arrays are shifted 1/2 of a pixel with respect to each other captured 
image. 

3. The apparatus of claim 1 having three fixed detector arrays. 

4. The apparatus of claim 3, wherein the three detector arrays are 
configured for capturing a red, a green, and a blue image, and wherein 
the images captured by the detector arrays are shifted 1/3 of a pixel 
with respect to each other captured image. 

5. The apparatus of claim 3, wherein the three detector arrays are 
configured for capturing Bayer pattern images, and wherein the images 
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P. 24 



captured by the detector arrays are shifted 1/3 of a pixel with 
respect to each other captured image. 
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6. A method of imaging an object comprising the steps of: 

(a) capturing three images* of the object; 

(b) shifting each captured image a subpixel amount with respect to the 
other captured images; and 

(c) interpolating between the shifted images to form a composite 
image. 
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7. A method of imaging an object comprising the steps of: 

(a) capturing two Images of the object; 

(b) shifting each captured image a subpixel amount with respect to the 
other captured image; and 

(c) interpolating between the shifted images to form a composite 
image. 
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